Skip to content

feat(prose-tests): record everything the agents do, results included - #557

Merged
leeovery merged 1 commit into
mainfrom
prose-tests/record-everything
Jul 27, 2026
Merged

feat(prose-tests): record everything the agents do, results included#557
leeovery merged 1 commit into
mainfrom
prose-tests/record-everything

Conversation

@leeovery

@leeovery leeovery commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • The gap this closes. The action log captured a command's input and never its result — and a walker exploited exactly that. It claimed "the snapshot's MENU section is empty", the asserter had no way to check, and four consistent runs across two cases agreed on something untrue. Running the gateway by hand showed a fully populated selection menu: the walker had auto-selected, then justified it by citing the emptiness of a different snapshot (gateway.cjs view, not the bare selection call).
  • So the hook now records everything: intent before each call (PreToolUse), the result after it with the output attached (PostToolUse), failures (PostToolUseFailure), and the finish (Stop). The logs are throwaway — they live in the disposable world and die with it — so there's no reason to record less.
  • All three agents are instrumented, each for what it can get wrong:
agent events why
prose-walker Pre / Post / Failure / Stop the full record of the walk, results included
prose-orchestrator PostToolUse (Bash) its own steps become visible — self-scopes via the world path in its commands
prose-asserter PreToolUse (any) it is contracted to use no tools; a call is a breach, logged to a repo-local file rather than vanishing
  • The asserter is now told the record settles claims about output: where its account of what a command returned disagrees with the record, the record wins and the discrepancy is itself a finding.

Test plan

  • npm test green: 1699 tests, 0 fail.
  • Hook unit-tested directly for both event shapes; the rendered view shows results inline:
  1. PostToolUse         Bash   cd . && node x.cjs
     → ok: === MENU === Which feature? - 1 Pay
  2. PreToolUse          Read   ./.workflows/x.md
  • Output truncated to 400 chars per entry to keep the asserter prompt manageable.
  • The two continue-* cases that failed on the false claim should now be judged against what the gateway actually returned — worth re-running.

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557 👈 current
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

This was referenced Jul 26, 2026
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 4790742 to 959a33a Compare July 27, 2026 14:32
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from d4c5490 to f09d886 Compare July 27, 2026 14:32
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 959a33a to 5701d7f Compare July 27, 2026 14:34
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from f09d886 to 499e06a Compare July 27, 2026 14:34
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 5701d7f to 97069a9 Compare July 27, 2026 14:37
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 499e06a to 5ff1980 Compare July 27, 2026 14:37
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 97069a9 to 7047929 Compare July 27, 2026 14:39
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 5ff1980 to 227b88d Compare July 27, 2026 14:39
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 7047929 to 75d4b1c Compare July 27, 2026 14:41
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 227b88d to cdc4a72 Compare July 27, 2026 14:41
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 75d4b1c to e50a157 Compare July 27, 2026 14:43
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from cdc4a72 to dab60ef Compare July 27, 2026 14:43
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from e50a157 to 1f8a100 Compare July 27, 2026 14:45
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from dab60ef to 94f3009 Compare July 27, 2026 14:45
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 1f8a100 to 613128f Compare July 27, 2026 14:47
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 94f3009 to 510908b Compare July 27, 2026 14:47
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 613128f to 0ebdd03 Compare July 27, 2026 14:49
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 510908b to 16f1664 Compare July 27, 2026 14:49
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 0ebdd03 to 5cf48e8 Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 16f1664 to 7222ed8 Compare July 27, 2026 14:51
@leeovery
leeovery force-pushed the prose-tests/consequence-claims branch from 5cf48e8 to 2c1fe91 Compare July 27, 2026 15:42
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from 7222ed8 to ffd9dfc Compare July 27, 2026 15:42
@leeovery
leeovery changed the base branch from prose-tests/consequence-claims to main July 27, 2026 15:43
The action log captured a command's input and never its result, and a
walker exploited exactly that gap: it claimed "the snapshot's MENU
section is empty", the asserter had no way to check, and four
consistent runs agreed on something untrue. Running the gateway by hand
showed a fully populated menu — the walker had auto-selected and cited
the emptiness of a different snapshot to justify it.

So the hook now records the lot. Intent before each call, the result
after it with the output attached, failures, and the finish. The logs
are throwaway — they live in the disposable world and die with it —
so there is no reason to record less.

All three agents are instrumented, each for what it can get wrong:

- the walker, on every tool event, into its world
- the orchestrator, on Bash, so its own steps are visible
- the asserter, on any tool call at all — it is contracted to use none,
  so a call is a breach and lands in a repo-local log rather than
  vanishing

The asserter is told the record now settles claims about output: where
its account of what a command returned disagrees with what the record
says came back, the record wins and the discrepancy is itself a finding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the prose-tests/record-everything branch from ffd9dfc to d5b6cfe Compare July 27, 2026 15:44
@leeovery
leeovery merged commit 70ab79c into main Jul 27, 2026
@leeovery
leeovery deleted the prose-tests/record-everything branch July 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant